Display and report generation platform for testing results

ABSTRACT

A data collection, display, and report generation platform has a first input interface configured to present a learning module comprising a series of questions and answers on a plurality of successive question/answer screens comprising a plurality of radio buttons, at least one of the radio buttons configured to accept both a first input action and a subsequent second input action, each providing a different visual indication. The first and second input action each indicate a different confidence level of a learner&#39;s answer. A display dashboard displays a plurality of data visualizations of metrics of misinformation and struggle of plurality of learners based on a plurality of answers collected through the first input interface, and comprises one or more bar graph displays, one or more heat map displays, and one or more sorting tools configured to alter the one or more bar graph displays or one or more heat map displays.

The present Application for Patent is a Continuation of PatentApplication No. 17/026,731 entitled “DISPLAY AND REPORT GENERATIONPLATFORM FOR TESTING RESULTS” filed Sep. 21, 2020, pending, which is aContinuation of Patent Application No. 15/853,104 entitled “DISPLAY ANDREPORT GENERATION PLATFORM FOR TESTING RESULTS” filed Dec. 22, 2017 andissued as U.S. Pat. No. 10,803,765 on Oct. 13, 2020, and assigned to theassignee hereof and hereby expressly incorporated by reference herein.

FIELD OF THE DISCLOSURE

Aspects of the present disclosure relate to a display and reportgeneration platform for organizational test results. In particular, butwithout limitation, aspects of the present disclosure relate to visualdisplays for easily identifying the most relevant data gathered from theresults of administered computerized tests, and the automatic generationof data reports and summaries therefrom.

BACKGROUND

In the field of computerized organizational testing and training, therehave been many advancements in how to improve the learning, retaining,and mastering of bodies of information by individuals within a largeorganization. As computerized testing and training systems haveadvanced, various aspects of both testing and training have become moreeffective due in large part to these systems being able to providecustomizable testing and training based on each individual learner'sneeds. It is well known that individuals each have different learningstyles, abilities, and existing states of knowledge on any particularsubject. It is also known that taking such differences into account foreach individual can improve rates of learning, retaining, and masteringinformation. Various educational techniques have been proven in studiesto enhance the learning, retaining, and mastering of information, butthe more such techniques can be customized to each individual, the moreeffective the techniques can be. When there are multiple individuals forwhom techniques need to be customized, the number of possible iterationsof tests and training becomes exponentially large very quickly.

Many organizations require dozens, hundreds, or thousands of individuallearners to learn similar material. Schools, healthcare organizations,governmental organizations, private commercial enterprises, and thelike, often require that a large number of learners be trained andtested across various time periods and geographical locations. Advancedcomputerized testing systems for such environments provide thiscapability by implementing large centralized databases of learningmaterial as well as algorithms for delivering customized content fromthese databases over distributed networks.

As these computerized testing systems advance and become capable ofdelivering more customized material to more learners, organizations areusing such systems more and more. As a result, individual learners whoare the subjects of these tests often desire greater efficiency in boththe test-taking process and the learning process in order to reduce thetime spent on each. Opportunities exist for the various educationaltechniques that enhance learning to be implemented in more intuitive,user-friendly, and efficient ways on a graphical user interface.

Another challenge that exists in this field is that because so manylearners can be tested and trained, and because each of the tests can becustomized and contain many data points, the volume of data generatedfrom the tests can be extremely large. Valuable information can begleaned from such data but it is often cumbersome to manipulate thatdata and derive anything useful. Therefore, opportunities exist toreport and display learner data to test administrators in new and usefulways.

SUMMARY

One aspect of the disclosure provides a data collection, display, andreport generation platform comprising a first input interface configuredto present a learning module. The learning module may comprise a seriesof questions and answers on a plurality of successive question/answerscreens of the first input interface. At least one of the successivequestion/answer screens may be presented based on answers on a previousquestion/answer screen, and at least one of the plurality of successivescreens may comprise a plurality of radio buttons. Each of the pluralityof radio buttons may be associated with an answer to a question, and atleast one of the radio buttons may be configured to accept both a firstinput action alone and the first input action and a subsequent secondinput action. The first input action causes the at least one of theradio buttons to provide a first visual indication of the first inputaction, and the subsequent second input action causes the at least oneof the radio buttons to provide a second visual indication of the secondinput action. The first input action and the second input action mayeach indicate a different confidence level of a learner's answer. Theplatform may also comprise a display dashboard configured to display aplurality of data visualizations of metrics of misinformation andstruggle of plurality of learners on one or more interactive screensbased on a plurality of answers collected through the first inputinterface. The display dashboard may comprise one or more bar graphdisplays, one or more heat map displays, and one or more sorting toolsconfigured to alter the one or more bar graph displays or one or moreheat map displays.

Another aspect of the disclosure provides a data collection, display,and report generation platform comprising a first input interfaceconfigured to present a learning module. The learning module maycomprise a series of questions and answers on a plurality of successivequestion/answer screens of the first input interface. At least one ofthe successive question/answer screens may be presented based on answersto a previous question/answer screen, and at least one of the pluralityof successive screens may comprise a plurality of radio buttons. Each ofthe plurality of radio buttons may be associated with an answer to aquestion, and at least one of the radio buttons may be configured toaccept both a first input action alone and the first input action and asubsequent second input action. The first input action causes the atleast one of the radio buttons to provide a first visual indication ofthe first input action, and the subsequent second input action causesthe at least one of the radio buttons to provide a second visualindication of the second input action. The first input action and thesecond input action may each indicate a different confidence level of alearner's answer. The platform may further comprise a learner dashboardconfigured to display a plurality of data visualizations of metrics ofmodule and course progress of a learner on one or more interactivescreens based on a plurality of answers collected through the firstinput interface. The learner dashboard may be configured to display afirst circular graph indicating module progress, a second circular graphindicating course progress, a time-spent bar graph comparing time spentby a learner to other learners, and an estimated time of completion.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary question-and-answer input interface of thepresent disclosure.

FIG. 2 shows another question-and-answer input interface display of thepresent disclosure.

FIG. 3 shows exemplary answer graphical displays of the presentdisclosure.

FIG. 4A shows a question-and-answer progress display of the presentdisclosure.

FIG. 4B shows a prompt given through an artificial intelligence-basedguidance system of the present disclosure.

FIG. 4C shows another example of a prompt given through an artificialintelligence-based guidance system of the present disclosure.

FIG. 5 shows an exemplary learning materials display interface of thepresent disclosure.

FIG. 6 shows an exemplary learning materials display interface of thepresent disclosure comprising image and video materials.

FIG. 7 shows a question feedback input interface of the presentdisclosure.

FIG. 8 shows an individual learner course dashboard comprising severalvisual information displays.

FIG. 9 shows an individual refresh and review interface of the presentdisclosure.

FIG. 10 shows a report home administrator dashboard comprising severalvisual information displays pertaining to multiple learners within acourse.

FIG. 11 shows a course summary administrator dashboard comprisingadditional data visualization displays.

FIG. 12 shows a module summary administrator display which is anavailable subcategory of the course summary administrator dashboard ofFIG. 11 .

FIG. 13 shows a learner progress administrator dashboard with sortablemenus for report generation.

FIG. 14 shows a lower portion list view of the learner progressadministrator dashboard of FIG. 13 .

FIG. 15 shows a course misinformation administrator dashboard displayingbar graph visualizations of quantified information from multiplelearners in a course.

FIG. 16 shows a detailed heat map data visualization display showingmisinformation by learners in a course, sorted by learner and topic.

FIG. 17 shows additional features of the heat map data visualizationdisplay of FIG. 16 .

FIG. 18 shows a course struggle administrator dashboard displaying bargraph visualizations of quantifies information from multiple learners ina course.

FIG. 19 shows a detailed heat map data visualization display showingstruggle by learners in a course, sorted by learner and topic.

FIG. 20 shows an actionable analytics report view generated by thesystem of the present disclosure.

FIG. 21 shows a logical diagram of the display and report generationsystem configured to implement aspects of the present disclosure.

FIG. 22 shows a logical diagram of computing devices that may be used toimplement aspects of the present disclosure.

DETAILED DESCRIPTION

A first part of the disclosure relates to user interface displays thatfacilitate the collection of learner input, increase learner engagement,increase learner information retention, and speed the learning andtesting process. The learning systems described herein pertain tolearning courses (which may be referred to as tests or assessments) thatimplement two-dimensional multiple choice question and answer sets(which may also be referred to simply as “two-dimensional questions”).The two dimensions refer to 1) a substantive answer and 2) a confidencelevel the learner has that the substantive answer is correct. For eachmultiple choice question, the selection of an answer is eithersubstantively correct or substantively incorrect. For each substantiveanswer a learner chooses, a learner's confidence level can be eithersure or unsure. As a result, four two-dimensional answers are possible;sure (confident) and correct; sure (confident) and incorrect; unsure andcorrect; and unsure and incorrect.

Organizations can make different value judgments on the importance of alearner's confidence in a correct or incorrect answer, but generally, itis possible to categorize the different types of answers as levels ofknowledge. The following paragraph describes one system of categorizinga learner's “knowledge state” as a function of the learner's substantiveanswer, level of confidence (if asked), and number of times a learnerhas responded in a particular way. In the present system ofcategorization, when a learner answers a question confidently andcorrectly once, that learner can be designated as having “proficient”substantive knowledge. When a learner answers the same or similarquestion confidently and correctly twice or more times, the learner canbe designated as having “mastery” over the substantive knowledge. When alearner answers a question correctly but is unsure, the learner can bedesignated as having an “informed” knowledge state. The learner may needto be asked the question again later to determine whether the learnerhas moved into a proficient state. An unsure and incorrect answer mayresult in the designation of an “uninformed” knowledge state. Aconfident and incorrect may result in the designation of a “misinformed”knowledge state. This knowledge state may also be referred to as alearner having “confidently held misinformation” throughout thedisclosure in order to highlight significant impacts of this knowledgestate within the testing and training systems of organizations. Some orall of these identified knowledge states may be used in the reportingand display systems of the present disclosure.

In the system of the present disclosure, some two-dimensional questionand answer sets may comprise one one-dimensional answer option that issimply “I don't know.” Such answers indicate a “not sure” knowledgestate, which may be considered slightly different from an “informed” or“uninformed” knowledge state from a pedagogical standpoint. Suchdifferentiation may be valuable to a test administrator for the purposesof facilitating learning. In sum, the different knowledge statesassigned to learners as a result of their answers in the system of thepresent disclosure may be categorized as follows:

-   -   1) Sure and correct two or more times: Mastery    -   2) Sure and correct once: Proficiency (or Mastery if on the        first try)    -   3) Unsure and correct: Informed    -   4) Unsure and incorrect: Uninformed    -   5) Sure and incorrect: Misinformed (or having confidently held        misinformation)    -   6) “I don't know” one-dimensional answer: Not sure        In order to simply the manner in which these knowledge states        are reported, the categories may be simplified into three        knowledge states, comprising “Mastery” for any number of sure        and correct answer, “Uncertainty” for any indication by the        learner of being unsure or not knowing, and “Misinformation” for        any sure and incorrect answer.

An aspect of the present disclosure pertains to facilitating theefficient capturing of a learner's one or two-dimensional answers. FIG.1 shows an exemplary question-and-answer graphical user interfacedisplay 100 (also referred to herein as an “input interface” on acomputer screen. The features shown and described throughout FIGS. 1-7are all part of a single “module” of testing and learning material. Ingeneral, a module comprises several distinct learning and testing topics(also referred to as “objects”). Each of these topics comprises severalquestions about the topic, each of which may be displayed or wordeddifferently than other questions about the same topic. The object alsocomprises learning material (e.g., an explanation, video, graphic, orother information) about the correct and incorrect answers to thequestions.

In FIG. 1 , a question 110 is displayed on the left side of the screenand four potential answers are displayed on the right. Of the fourpotential answers, the top three are two-dimensional answers 120, andone one-dimensional answer 125 of “I don't know yet.” The “I don't knowyet” option exists because the questions in the module being presentedin FIG. 1 implement a technique known as “priming.” This feature asks“priming” questions of a learner even before presenting any learningmaterial. Education research has shown that asking questions of alearner, even if the learner has no prior knowledge of a particular subject, primes the brain for receiving and retaining information when thelearning material is later presented. The learner can select theone-dimensional answer in this mode in order to speed the learning andtesting process, which will be described in more detail later in thisdisclosure.

The input interface 100 also comprises a pop-up instruction box 130,which appears when a learner first clicks on any of the one ortwo-dimensional answers 120, 125. Many users of computerized testingsystems are accustomed to clicking on one-dimensional answer choices tomultiple-choice questions, but many may be unfamiliar with systems thatreceive both substantive answers and confidence levels. There are anumber of possible ways a user could select both a substantive answerand a confidence level, which have been implemented in prior systemswith graphical user interfaces. For example, a user could select asubstantive answer out of one column and a confidence level out ofanother, thereby clicking on two radio buttons to indicate the twodimensions of the answer. As another example, a user could drag and dropsubstantive answers from one area of a screen into areas of associatedwith confidence levels on another area of a screen. The answer-receivingfunctionality and display of the present disclosure provides advantagesover these previously implemented methods.

As shown in the pop-up instruction box 130, a user may be presented withone large radio button 132 per answer, which may display differentvisual indicators based on the number of clicks and the type questionwith which the radio button is associated. If the radio button isassociated with a two-dimensional question, such as questions 120, andthe user clicks once, the radio button may display as half-full ofcolor, as shown in the half-full radio button 134. If the use clickstwice, the radio button may display as fully colored-in, as shown in thefull-color radio button 136. This feature gives a user a subtle cluethat there is more to just the substantive answer when first clicking onit. The second click requires the user to actively confirm that he orshe is sure of an answer. An advantage to this click and display systemis that a user does not have to draw his or her attention away from asubstantive answer and separately think about their confidence levelindependently of the substantive answer. That is, the user can remainfocused on the substantive answer while considering their confidence asa function of clicks. This input mechanism is the simplest kind of inputpossible into a computer screen, and is beneficial because it is fasterand more intuitive than dragging and dropping or clicking on twoseparate buttons.

There are particular advantages to the input display and mechanism ofthe present disclosure over existing methods in the art. By integratingthe confidence measurement with the action of selecting a question, thelift a learner gets from assessing confidence is in place. Previousmethods have implemented input displays and mechanisms that the ask thelearner of his or her confidence after he or she made a selection on theanswer (in other words, it is a 2 step process). That method presents aparticular problem: when a learner assesses their confidence after adecision is made, the confidence selection does not cause the learner toquestion their selection or review the other answers as part of theaction. The weighing of all the answers is where the learning liftoccurs. A learner really has to think about why they chose a particularanswer, and why not the others. This increases the priming effect whilehelping to drive deeper knowledge on the subject by including the wronganswers as part of what is being learned.

For one-dimensional answers, such as the “I don't know yet” answer 125,the radio button may immediately display a fully colored-in button 140upon being clicked just once, visually indicating to the user that thisquestion only has one dimension and is solely an indication of aknowledge state. This feature allows users to quickly distinguishwhether they are done providing a response to a question.

Another feature of this question-and-answer display 100 is shown in FIG.2 . A user may select up to two choices if he or she is unsure of theanswer. As shown, if a user has clicked on a two-dimensional answerchoice only once, and the radio button 210 is only half colored-in, theuser may select another radio button 212 once. In some embodiments, if auser has selected one of the two-dimensional answers as “sure” (e.g.,clicked it twice), the other answer choices may be rendered unclickable.This feature ensures that a user cannot mistakenly choose one answer as“sure” and another answer as “unsure” or “sure.” If a user does want toselect more than one answer, the user has to consciously confirm that heor she is truly unsure of both answers.

Additionally, the question-and-answer display of the present disclosurepresents a written confirmation cue of the user's indicated knowledgestate by displaying the words “I am unsure” 214 above the displayedsubstantive answer 216. The combination of the half colored-in radiobuttons next to the written words “I am unsure” in response to one clickin each radio button, before clicking to submit an answer, gives users avisual, written, and kinesthetic cue to reinforce the acknowledgement ofa confidence level. It is known that different learners process visual,written, auditory, and kinesthetic information differently, with one ortwo types of information being preferable to any particular learner.Having multiple types of cues associated with one piece of informationensures that more people can quickly understand the informationpresented on an interface.

Within the pre-learning questioning phase, an answer explanation may beimmediately presented to a learner after he or she submits a choice orchoices. FIG. 3 shows exemplary answer explanation displays 310, 320,330, and 340. In each of these displays, the answer choice the learnerselected is visually marked to indicate both the correctness orincorrectness of the substantive answer and the knowledge state of thelearner. In the first display 310, an unsure and correct answer 312 isdenoted by a half colored-in radio button with a vertical line down thecenter and the right half of the circle filled in. In the second display320, an unsure and incorrect answer 322 is denoted similarly to theunsure and correct answer 312, but it is displayed in a different color.For example, the unsure and correct answer 312 may be green and theunsure and incorrect answer 322 may be red, corresponding to manylearner's existing associations of green with a positive result and redwith a negative result.

In the third display 330, a sure and incorrect answer 332 is denotedwith a large “X,” which may be red in embodiments. In the fourth display340, a sure and correct answer 342 is denoted with a large check mark,which may be green in embodiments. The correct answer is not shown ifthe learner has selected the incorrect answer, and no supplementarylearning information is presented to the learner at this point. Instead,the learner's next step after answering a priming question is simply tomove on to the next question. This sequence of events is purposeful,because in the background, an algorithm uses the learner's answers fromthe priming questions as inputs to create a customized order of repeatand/or related questions within the module (as described previously, amodule may present questions that are exactly the same asearlier-presented questions, or it may present differently-worded orarranged questions about earlier-presented subject matter). The order ofa learner's answers may be referred to as an “answer sequence”throughout the disclosure. Rather than just answering priming questions,the learner will eventually move on to a phase in the module whereinpriming questions will be interspersed with learning material. Thisalgorithm for how various questions are presented will be described inmore detail later in the disclosure.

Another feature of the module algorithm is that if a user answers aquestion as sure and correct before the learner has seen any learningmaterial within the module, the answer counts as a “mastered” concept,which grants the learner progress toward the completion of the entiremodule. This feature allows a learner to complete a module more quicklyand efficiently if the learner comes in with existing knowledge of thesubject matter. A goal of the learning module algorithm is to get thelearner to answer questions as sure and correct as quickly as possible.FIG. 4A shows a visual completion bar 410 at the top of the question andanswer display 400. The completion bar 400 shows a learner severalaspects of their progress so far. As shown, the completion bar 400comprises several vertical marks representing total concepts that needto be completed within the module. The first vertical mark 411 in thisexample is filled in completely (e.g., as dark green) to represent thatthe user has mastered a concept. The dialog box 420 in the figure saysthat the learner has just completed their first question; in this case,the filled in vertical mark 411 and dialog box 420 indicate that thelearner has either 1) answered the question as sure and correct uponfirst seeing it, or 2) has answered the question sure and correct atleast once after previously answering otherwise. There are a number ofpossible combinations of answers and answer sequences that can lead to avertical mark being filled in completely. For example, a learner mayanswer a question any number of times as sure and incorrect, unsure andincorrect, or unsure and correct. Depending on a learner's previousanswers, the algorithm may require a learner to answer the same questiononce or twice as sure and correct. A learner must show mastery over eachpiece of learning material in a module, which would be represented byeach vertical mark in the completion bar 410 being filled in completely.

The completion bar 410 also indicates how much material in the modulethe learner has made satisfactory progress toward by partially shadingother vertical marks. As shown, the partially shaded vertical marks 412are lighter in shade than the fully shaded vertical mark 411 (e.g., theymay be a light green instead of a dark green). These partially shadedmarks may indicate that a learner has made some sort of progress, whichmay comprise one of several possibilities. For example, a learner mayhave answered a question as unsure and correct one or more times. Asanother example, a learner may have answered a question as unsure andincorrect once, but then subsequently answered the same question as sureand correct once. Or a learner may have answered as sure and incorrectonce, and then unsure and correct once. Several additional combinationsof answers and subsequent answers may result in partial shading ofmarks, but in general, partial shading of marks indicates some sort ofpositive progress or improvement over a learner's previous knowledgestate. Notably, an answer of “I don't know yet” results in a partiallyshaded mark, and is considered positive progress toward completion ofthe module. This feature encourages a learner to admit when they trulydo not know an answer instead of guessing. The learner is rewarded witha partially-filled in mark and is presented with learning material aboutthe question before being presented with it again. The next time thelearner sees the question, if the learner answers as sure and correctonce, the mark will be filled in completely.

The completion bar 410 also indicates how much material a learner hasanswered incorrectly and on which he or she has not yet made positiveprogress via a differently-colored vertical mark 413 on the right of thepartially-shaded marks 412. In some embodiments, thisdifferently-colored vertical mark 413 may be red. The mark 413 mayindicate some sort of negative result; for example, that a learner hasanswered a question as unsure and incorrect once or more time, or sureand incorrect once or more times. Such answer sequences may becharacterized as negative results of “misinformation” or “struggle,”which will be discussed in greater detail later in the disclosure. Uponanswering another question on the same subject matter correctly later,the red mark may change to a light green mark, and upon sufficient sureand correct answers to the same subject matter (e.g., one or two), thelight green mark may change to dark green.

Subject matter not yet seen by a learner in a particular module isrepresented with blank vertical marks 414 that have yet to be presentedto the learner. These blank vertical marks 414 will be filled in asdescribed above with dark green, light green, and red marks as thelearner progresses. Another feature of the progress bar 414 is that thedark green marks 411 will be accumulated from left to right, the lightgreen marks 412 will be filled in to the right of the dark green marks411, and the red marks 413 will be filled in on the right of the lightgreen marks 413. Any of the vertical marks of any color may be added orchanged in response to a learner's answers.

The completion bar method and display therefore provides a number ofadvantages to a learner. First, it provides a way for a learner tovisualize how much total module progress he or she has made and how muchremains. Providing a reliable visual indicator of progress has beendifficult in prior confidence-based testing platforms because theabsolute number of questions varies for each learner based on thelearner's pattern of answers. In the display of the present disclosure,the total number of subject matter concepts in a module is representedby the vertical hash marks, and they are adapted to reflect progress nomatter how many total questions the learner actually ends up answering.A second advantage is that it provides a way for learners to visualizethe quality of their progress. That is, a user can see a difference inprogress made on the completion bar 410 between rushing throughquestions and perhaps answering many of them incorrectly or withinaccurate confidence assessments, and carefully evaluating theirknowledge state. The difference becomes apparent when a user sees eitherthe light green and subsequently dark green vertical marks increase, orthe red marks increase. A third advantage is that it allows the user tovisually identify the efficiency of their learning. It does not takelong for a new learner to the system to associate increased red markswith increased time required to complete the module. This display methodtherefore teaches a learner very quickly (even one who is unfamiliarwith the system of confidence-based learning) the importance ofaccurately assessing their own knowledge state.

FIG. 4A also shows a feature associated with another aspect of thedisclosure, which is an artificial intelligence (AI)-based learnerguidance system. In FIG. 4A, a dialog box 420 is displayed to a learnerafter completing a first question to draw the learner's attention to thecompletion bar 410. This feature assists the learner by guiding him orher on tips to complete the test more efficiently. FIGS. 4B and 4C showother exemplary prompts in dialog boxes 430 and 440. The dialog box 430displays a message saying “Please answer with honesty. if you are unsureor don't know, say so.” This prompt is based on an algorithm thatassociates particular phrases with learner answer sequences. Forexample, the prompt in dialog box 430 may be shown in response to alearner getting several answers in a row “sure and incorrect”. Suchprompts may be programmed to appear because certain patterns indicatethat something can be reasoned from the pattern. In this case, thechances that a learner is really sure and correct for too many questionsin a row likely indicates that the learner is not being truthful abouthis or her knowledge state. For the prompt in dialog box 440, which says“you have been doing great in this module, keep it up . . . ,” such aprompt may be in response to a learned pattern by the guidance systemalgorithm that shows that learners at this particular point in themodule likely need encouragement to finish in a timely manner.

A range of prompts can be used within the AI guidance system, whichmimics an intelligent response that an instructor might give if theinstructor were familiar with each answer and knowledge state given bythe learner in real time. These prompts can be used to discourage“cheating” when cheating patterns are detected. In many embodiments ofthe learning system, a learner is subject to a “penalty” of having toanswer a question twice as “sure and correct” after getting an answer“sure and incorrect” or selecting two answers as “unsure.” A learner maybecome aware of this time penalty and try to “cheat” to save time. Inthis learning system, “cheating” may be defined as a user expresslytrying to avoid the penalty.” For example, one way to cheat is to answer“I don't know” or “unsure on just one answer” to all the questions,write down the answers as they appear, and answer them correctly thenext time through without actually learning. Since the point of thelearning system is for users to assess their knowledge states correctlyand actually learn, prompts may appear upon the detection of cheatingpatterns that say “accurately estimating your confidence helps yourmemory and saves you time. You'll spend extra time where you'reconfident and wrong.” Other prompts may be generated and displayed tousers in response to detected answer patterns that show that a learneris very smart, has great metacognition, has poor metacognition, or hashigh levels of confidently held misinformation. Other prompts thatencourage a learner to stay on task or to improve may be used in variousembodiments. The timing of when each of these prompts will bepersonalized to each learner, since each individual learner can have adifferent question and answer sequence while proceeding through themodules. Having a uniquely-tailored AI guidance system that responds tolearners as they need particular prompts further enhances theeffectiveness of teaching concepts to learners of all types.

Turning now to FIG. 5 , shown is a learning display 500 of the presentdisclosure in which learning material 510 is displayed under aquestion-and-answer review section 520. The system of the presentdisclosure automatically presents learning material in this manner aftera certain number of administered questions; in some embodiments, thislearning material is presented after the learner has answered five toten questions. This learning material, however, is only presented to alearner for questions to which the learner has recently answered “Idon't know yet” or incorrectly. As shown in the question-and-answerreview section 520, the learner had answered the question as “sure andincorrect.” In different display sections of the computerized testingsystem of the present disclosure, the learner may optionally reviewlearning material for questions that they answered correctly.

FIG. 6 shows additional types of learning material that may be displayedto a learner as a part of the module. Some learning material may includeimages as shown in the upper learning display 610, and some learningmaterial may include videos as shown in the lower learning display 620.As discussed briefly earlier in this disclosure, questions and answersthemselves about particular subject matter may vary in wording andpresentation order when shown to a learner more than once. In a similarmanner, learning material about particular subject matter may includevary in visual presentation to a learner if it is shown to a learnermore than once. For example, a first time a learner sees a particularlearning segment, it may comprise only written information, but if thelearner encounters it again, it may also comprise a photo or a video. Itis contemplated that other types of audio or visual learning materialsmay be included, such as charts, graphs, sound recordings, or the like.As will be discussed later in this disclosure, many data points may begathered by the computerized system of the present disclosure over manytests and many learners. Different types of learning material may bepresented in order to evaluate, over large data sets, which type oflearning material is most effective for most learners. In someembodiments, this information may be used to refine the learningmaterials to maximize learner efficiency across large organizations.

FIG. 7 shows a feature that can also be used to refine questionsthemselves within modules. Because the testing, training, display, andreporting systems of the present disclosure can be used across variousindustries for various topics, many different individuals will beresponsible for creating the questions, answers, and learning materialscontained in the modules. The feedback input form 700 of FIG. 7 allows alearner to provide feedback about a particular question to facilitatethe improvement of questions.

Another aspect of the disclosure comprises report and display systemsfor conveying meaningful information to test takers, supervisors, andadministrators in easy-to-understand formats. FIG. 8 shows an exemplaryprogress displays for test takers. As shown, a progress summary display800 shows a test-taker, through a circular graph 810, how much of acourse a learner has completed. It also shows an estimated time tocomplete segment 820. This estimated time may be based on the time ithas taken the learner so far and identifying patterns that the learnerhas shown so far and may be likely to demonstrate in the future.Alternatively, it may be based on how many questions are left and anaverage time of completion of a sample number of other learners.Directly beneath that segment 820 is a course overview comparison 830between the test taker and other test takers. The progress summarydisplay 800 provides several advantages for learners, such as allowingthem to stop taking a module after making partial progress and quicklyunderstanding where they are upon returning. Another advantage is givingthem an estimate of time to complete, which may provide motivation tocontinue and perhaps improve. As discussed previously, several visualcues and features incentivize learners to accurately assess theirperformance to reduce time completing module. The course overviewdisplay 830 shows learner how he or she is doing compared to his or herpeers, which may provide motivation either positively or negatively.

An individual learner's course dashboard 850 may show comparisonsbetween the learner and all other learners using a line graph format860. It may also include a separate course progress circular graph 870,which may help a learner visualize how many modules in the course arenot started, started and in progress, and completed. A module table 880below provides information on each of the modules. Though only onemodule is shown in the table 880, it is contemplated that in somecourses, learners may have to take dozens of modules, so keeping trackof progress of how many modules are complete, and detailed informationabout the progress within each one, may be convenient to a learnercompleting a large course over a long period of time.

FIG. 9 shows another aspect of the learning system, which is a refreshand review interface 900. Here, a learner can access, for each module, a“smart refresh,” “refresh,” and a “review” option. Each of these modesis designed for a learner who has completed at least part of the moduleonce and who needs to re-learn or reinforce the information in part orin whole. A learner may do this to study for an upcoming test or becausethese re-learning courses are assigned to them for job trainingpurposes. A “smart refresh” mode 910 (which may also be referred toherein as a “smart refresher”) is a mode in which a learner can retakejust the portions of the module that he or she did not answerconfidently and correctly immediately during the learner's first moduletry. That is, the smart refresh mode does not spend time on anythingthat the learner already knew prior to the first module try. The“refresh” mode 920 (also referred to herein as a “full refresh”) is afull retaking of the module. The “review” mode 930 allows a learner toreview the full list of questions, learning material, and correctanswer. This review mode may only be available to a learner who hascompleted the entire module once, which provides an incentive for thelearner to complete it.

Another aspect of the learning and display system comprisesadministrator displays for multiple learners. The displays of thepresent disclosure are designed to help supervisors or anyone else incharge of groups of learners identify two important metrics:misinformation and struggle. Misinformation, also known as confidentlyheld misinformation, is the knowledge state wherein a learner is wrongbut is sure that he or she is right. Confidently held misinformation hasbeen identified as the most costly and dangerous knowledge state,especially in professional fields where life and health are at stake.Consider, for example, professionals in the medical field, or aviation.These individuals go through extensive training and must make manyimportant decisions throughout each day. People's lives depend on themknowing correct information, and those professionals know that. It isimpossible for humans to know the correct information they need to know100% of the time, but if an individual knows they don't know, or isunsure, of a correct answer, that individual is highly likely to look upthe answer or seek help from someone else. However, if the aviation ormedical professional is wrong but is sure that they are right, they areunlikely to ask anyone for help, and will likely proceed with a wrongdecision. In such cases, confidently held misinformation can result ininjury or loss of life. Even in fields where mistakes do not result inphysical harm, confidently held misinformation can be costly in terms ofmoney and productivity.

Previous learning systems lacked efficient ways to identify confidentlyheld misinformation, if it was even identifiable at all. The displaysystems of the present disclosure allow supervisors and testadministrators to identify such misinformation quickly and accurately atthe level of individuals or across segments of an organization.Throughout the disclosure, the terms “supervisor,” “administrator,” or“manager” may be used somewhat interchangeably to refer to someone whohas access to view the results for groups of particular learners.Knowledge about misinformation is invaluable; if a supervisor canidentify a particular individual who has high levels of confidently heldmisinformation, or business units where an unusually high number ofindividuals had misinformation about the same topic, targetedremediation can be implemented immediately. As previously described, thequestion display and answer feedback system of the present disclosurehelps learners assess their own confidence levels accurately, whichcaptures when a learner has confidently held misinformation. Then, thedisplay and reporting system identifies who, within an organization, hasthat misinformation, and what topic the misinformation concerns.

As mentioned, the reporting and display systems also show how muchstruggle was involved for a particular learner or a particular topic.The metric of struggle is defined by instances wherein a learner getsanswers incorrect more than once. There are several variations of answerpatterns that can fall within the definition of “struggle” in thissystem. For example, a learner might answer confidently and incorrecttwice; or unsure and incorrect twice; or incorrect, correct, thenincorrect (along with any possible knowledge state). In contrast, alearner's ease of progress can be defined as “learned it” in instanceswherein the learner gets an answer incorrect once, but subsequently getsanswers on the same topic correct without any problems. Alternatively, alearner's ease of progress may be defined as “knew it” for instanceswherein the learner got all answers to a topic correct (even if one ofthose answers was with an unsure knowledge state). Measuring ease ofprogress in this way provides administrators an additional dimension toevaluate and address. When supervisors can easily identify that aparticular learner struggled significantly more than others, or thatmany learners struggled on a particular subject, the supervisor canfocus attention on those learners and those topics. Organizationalproblems can be solved more effectively when such insight is availablethrough easy-to read data visualizations, rather than text or numericaltables. Beneficially, outliers can be detected at a glance, which areoften the most relevant kinds of information a test administrator orsupervisor needs.

FIG. 10 shows a “report home” administrator dashboard 1000 that providesoverviews of courses taken by multiple leaners. An administrator mayoversee several different courses, and may choose which reports to seein a course selection menu 1005. A course overview section 1010 showsnumerical data including average progress of all learners through thecourse, average completion time, a percentage of misinformation, anumber of refreshers taken, a number of total learners, and a number oftotal modules. A learner progress section 1020 has a circular graph 1025showing a number of learners not started, a number of learners inprogress, and a number of learners completed on the course. A coursemisinformation section 1030 shows, in horizontal bar graph formats, thepercentage of misinformation (e.g., in red), uncertainty (e.g., inyellow), and mastery (e.g., in green) separately for each module. Somecourses have many modules, and outliers of misinformation would beeasily visible in this section. A course struggle section 1040 shows, ina vertical bar graph format, the ease of progress of the learners intotal. In the example in FIG. 10 , 4% of learners struggled, 51% learnedit, and 45% knew it. The course misinformation section 1030 and coursestruggle section 1040 show high-level visual summaries of these metrics.It is contemplated that the sections shown on this report home dashboard1000 may be customized to show the most relevant summary informationthat an administrator wants to view. Views on subsequent displays willshow these metrics in greater detail.

FIG. 11 shows more detailed data visualizations of a particular coursein a course summary administrator dashboard 1100. A “course average”section 1110 shows text and numerical averages and totals of variousmeasures of the course and what learners have done in the course. Alearner progress section 1020 shows a learner progress section 1120,which may be repeated from the report home dashboard 1000 of FIG. 10 .The learner progress section may be useful to have at a glance on thisdisplay because the relevance of the other visual information may be afunction of how many learners are in progress or have yet to take themodule. For example, if only one or two learners has begun or completedthe course, the sample size may not be big enough for the administratorto derive any conclusions about the course. However, if, as shown inFIG. 11 , if more than half of the total learners have begun orcompleted the course, the information about the course and the modulebecomes sufficient to identify patterns and outliers.

A “most challenging module” section 1130 shows bar graph comparisons ofeach module in the course with a calculation of “average NPA” in themodules. This simply shows which module had the most instances ofinformation, and may be presented as a list or other visual in otherembodiments. A “module progress” section 1140 shows average progress,completion time, last activity, and average knowledge of learners duringthe different variations of each module. The variations include alearning mode, full refresher, and smart refresher, as explained earlierin the disclosure. As shown, an “average knowledge” metric 1145 may beshown by a horizontal bar graph. The bar graph shown represents aroll-up (i.e., average) of the misinformation, uncertainty, andmisinformation of all the learners who have completed the module. Theindividual reports on the learners whose scores are represented by thisaverage are shown in FIG. 15 . It is an expected result is that learnerswho took the smart refresh variation had higher average knowledge (suchas 100% mastery) because they had previously completed the coursesuccessfully.

From the course summary dashboard 1100, an administrator can choose toview additional information about individual modules. An administratorcan select a variation of Module 1, such as the learning module 1150 toview additional details about the learning module, as shown in FIG. 12 .

FIG. 12 shows a view that is available when an administrator clicks onthe learning module link 1150 shown in FIG. 11 . This view shows amisinformation (knowledge) column 1220 and a struggle column 1230 foreach question in the module, which are listed in a question column 1210.This view allows a detailed look at particular questions within a modulethat may be difficult for many learners in terms of eithermisinformation or struggle. For example, the seventh question on thelist 1217 shows an unusually high amount of misinformation across manylearners, as does the twelfth question on the list 1222. This view alsoshows that even though this is a question that identifies high levels ofmisinformation, the same learners do not struggle much to learn thecorrect answer, as shown in the corresponding struggle graphs 1227 and1232. Such information may be important to an administrator. With thisknowledge, the administrator can know that future learners may be likelyto have similar misinformation, but that it is easy to correct. This mayimpact an organization's strategy for teaching such things early andoften to new hires, for example.

An administrator can also change the view of information in this modulesuch that misinformation and struggle is viewable on alearner-by-learner basis instead of a question-by-question basis. Suchinformation gives additional insight into whether particular learnershave high levels of information and This view can be selected on thescreen by clicking on the “learners” portion of the sorting feature1240. Turning back briefly to FIG. 11 , each of the variations ofmodules may be selected in to view the information shown in FIG. 12 byquestion and by learner. The administrator may select the full refresherand smart refresher module variations and see relative levels ofknowledge and struggle in each. In most situations, the misinformationand struggle for those variations should be lower than for the learningmodules, but the views allow the administrator to identify whether thatis not the case. Each of the display features allow quick identificationof important granular details that could be easily missed if they onlyexisted in text form.

FIG. 13 shows a learner progress dashboard 1300 that provides a view ofthe progress of sets of learners through courses. This dashboard 1300view is sortable by a “view-by” drop-down menu 1310, which in thisembodiment shows that the circular graph 1320 can be sorted according tototal learners to whom the course is available, functional area, hiredate, and hometown region. It is contemplated that other sortingcriteria may be implemented as well. The dashboard 1300 view can also besorted by other criteria through the “assignment type” drop-down menu1330 and the “begin date” and “end date” calendar menus 1340 and 1350.The assignment type drop-down menu 1330 may allow an administrator tochoose from the learning mode, full refresh mode, and smart refresh modeto further refine the display criteria. As shown, when an administratorselects a view of particular types of learners from the view-bydrop-down menu 1310, the administrator can see on the circular graph1320 how many learners have completed, started, and not started thecourse.

These views are beneficial for the various ways many organizationsconduct their learning and testing operations. In many workplaces, forexample, it is impractical to schedule all employees, or even groups ofemployees, for testing at the same time like in school. Employees mayneed period of several weeks or months to complete learning and testingmodules, depending on various factors such as their complexity and theamount of continuous learning required in a particular field. Whentesting is conducted over a period of time, this learner progressdashboard view can help an administrator identify several things thatmay be occurring within a group of learners. For example, theadministrator may be able to see whether a significant number oflearners procrastinate, whether learners take a long time to completethe courses once they start, and, whether enough learners have taken acourse to derive meaningful conclusions about the content. By sortingthe learners by other criteria, such as hire date, an administrator canview whether all new hires—an especially important group to monitor,have timely completed a course. By viewing learners by functional groupor geographic area, an administrator can identify patterns and recognizewhether a particular functional group or geographic area isexceptionally fast or slow at completing modules, which providesopportunities for administrators to offer targeted praise or correctiveefforts. Identifying such patterns without the visual display shown inthe learner progress dashboard 1300 would be difficult and cumbersome,even with the sorting features available on spreadsheets, because of themultiple dimensions of information conveyed in these views (i.e., typeof learner, type of learning mode, date, and three categories of courseprogress). The display on the learner progress dashboard 1300 overcomessuch difficulties.

FIG. 14 shows a list view 1400, which is on a lower portion of thelearner progress dashboard 1300 shown in FIG. 13 . The list view 1400shown in FIG. 14 is organized alphabetically by learner, whichcorresponds to the “learner” selection in the view-by menu 1310. Each ofthe columns in this list view 1400 can be used to further sort accordingto the column criteria. The list view 1400 also changes to correspond tothe other possible selections in the view-by menu 1310, such that thefirst column 1410 may alternatively show a list of functional area, hiredate, or geographical area, for example.

FIG. 15 shows another aspect of the display system of the presentdisclosure, which is the course misinformation dashboard 1500. A mainmenu side bar 1510 is shown on the left of the screen, as it has been inother figures. Many computer users are familiar with menu optionsdisplayed in this manner, which can be used to guide a user to importanttopics of interest available to the user. A unique feature of thedisplay system is that the concepts of “course misinformation” and“course struggle” are such important topics of interest that can bedisplayed to a user in a main menu. In these sections, the dashboardscan take complex information gathered from learners the learninginterface input displays and distill it into useful, intuitive, andactionable visual feedback.

The course misinformation dashboard 1500 provides several ways for anadministrator to view the important metric of misinformation accordingto several criteria. Similar to the learner progress dashboard 1300, thecourse misinformation dashboard allows the administrator to sort theview by learner, functional area, hire date, and geographical region, aswell as by assignment type, date range, and course. The informationdisplayed here is a misinformation graph 1550. In this view, knowledgestates of learners are grouped into three categories, which aremisinformation, indicated in one color (red, for example), uncertainty,indicated in another color (yellow, for example), and mastery, indicatedin another color (green for example.) As discussed previously, severalpossible patterns of answers may result in the designations ofmisinformation, uncertainty, and mastery, respectively. For example, themisinformation designation may be assigned to a learner about aparticular question (or module or topic) if a learner answered twosimilar questions “sure and incorrect”, or answered “not sure” andincorrect on one question and then “sure and incorrect subsequently.” Inother words, multiple different answer combinations may result in thedesignation of “misinformation.” One advantage to distilling manypossible answer combinations into just three designations is that havingthree categories allows the administrator to focus on the most criticalinformation. Other, more detailed category breakdowns with additionalcolor representations are available in other views. However, it isimportant for administrators to be able to recognize misinformation at aglance.

The misinformation bar graph 1550 can be further sorted according tobest and worst performers from a graph sorting drop-down menu 1520. Themisinformation bar graph 1550 shows a view sorted by worst performers.As shown, the worst performer bar graph 1551 is shown as the one withthe greatest total uncertainty and misinformation together, but is notnecessarily the one with the most misinformation alone. In variousembodiments, the calculation of “worst performer” may be either the oneshown (with the highest combination of uncertainty and misinformation)or the one with the most misinformation. Different algorithms may beused to weight and calculate different rankings of performance. Asshown, in this particular graph, the second worst performer bar graph1552 shows a learner with the high amount of uncertainty andmisinformation, but who performed worse than the learner shown by thethird worst performer bar graph 1553, even though that learner had moremisinformation. This may be because in the algorithm used to calculatethe results shown weights very high amounts of uncertainty higher,resulting in a “worse” performance. However, in other embodiments, theselearners may be sorted by “most misinformation”

Another feature that is available in this misinformation bar graph viewis that an administrator may hover over sections of the bar graph andview a numerical percentage represented by the section. For example, apop-up bubble may be displayed when a cursor is over a yellow“uncertainty” section to display the percentage of uncertainty and thename of the learner whose answers are represented by the graph. Thispop-up numerical percentage display may be available in other displays,including the bar graph and the heat map display discussed below.

The course misinformation dashboard 1500 can also display misinformationdata in a more detailed, but still intuitive format. Turning now to FIG.16 , shown is a “heat map” display 1600 that illustrates levels ofmisinformation by topic. An administrator can use the heat map displayoption icons 1610 and the bar graph display option icon 1620 to switchbetween the views shown in FIGS. 16 and 15 . Additionally, anadministrator can use selector option 1630 to switch between showingentire modules or showing topics in this heat map display 1600. The heatmap display 1600 shows levels of misinformation by topic, with acolor-coded misinformation level key 1650. Each topic represents aroll-up of several questions within the topic. This key 1650 shows arange of colors from green to red, as an example, each associated withcertain percentages of misinformation. As shown, percentages ofmisinformation are represented by different colors at 10%, 20%, 30%,40%, 50%, and greater than 50%. In the misinformation bar graph 1550shown in FIG. 15 , there were only 3 categories spanning a larger rangeof percentages, but the heat map display 1600 shows more detailedpercentage break-downs. In the heat map view 1600, which is sorted by“best performers” and topics, several pieces of information areavailable to an administrator at a glance. For example, an administratorcan see that even the worst performers only had high levels ofinformation on one or two topics. An administrator can also see thatmost of the misinformation among all learners was concentrated over twotopics, which may indicate that particular subject matter or particularquestions about that subject matter were especially difficult for mostlearners.

FIG. 17 shows additional features available in the heat map view. Theheat map display 1700 is sorted by worst performers and topic. In thisview, there are more topics shown than in the heat map display 1600 ofFIG. 16 , and many data points are visually represented. Anadministrator can tell at a glance which topics were the subject of themost information, and which learners performed the worst, but additionalvisual cues assist the administrator in identifying more detailedinformation. When an administrator hovers over a particular block in theheat map, a pop-up numerical percentage 1705 may appear, which displaysthe percentage of misinformation. As described earlier, red canrepresent any percentage over 60%, so it may be useful to see the actualpercentage within that range. Additionally, the pop-up 1705 may show thelearner's name and topic so that an administrator can easily view thisinformation in the midst of many data points represented by thecolor-coded blocks. Another feature is that the learner's name and topicmay be highlighted with a row highlighter 1720 and column highlighter1710 to further assist the administrator's view. This heat map display1700 allows administrators to look down particular rows in order toidentify poor performers and down particular columns to identifydifficult topics. Administrators may wish to de-clutter this display byremoving some visual indicators. An administrator can do so by clickingand dragging an indicator 1720 along a show/hide adjustment bar 1725 (aduplicate version 1730 and 1735 is shown at the bottom of the screen).clicking and dragging the indicator across the bar results in certainpercentage ranges to be shown or hidden on the heat map display 1700.For example, if an administrator drags the indicator 1720 to a spot onthe adjustment bar 1725 that corresponds with the 40% mark, only theblocks representing misinformation above 40% will be shown on the heatmap display 1700, and the rest of the blocks will be hidden. Theadministrator may drag a second indicator 1728 from the opposite side ofthe adjustment bar in order to limit the percentage range from the upperbounds. The adjustment bar feature allows administrators to see specificranges of performance, which may assist them in making changes in futureversions of a test. For example, if an administrator is looking tostreamline future tests, he or she may select a range of onlymisinformation of 10-20% to quickly identify topics that learners rarelyhave misinformation, and eliminate those.

FIG. 18 shows a course struggle administrator dashboard 1800. Thisdashboard 1800 is similar to the course misinformation dashboard 1500,but allows an administrator to view struggle, which is the other mainmetric of the display system. “Struggle” is a measure of non-progressattempts, which may encompass anything other than a confident andcorrect after seeing the correct learning material at least once. Asshown, the course struggle dashboard 1800 has a struggle bar graph 1850that places measurements in one of three categories, the categoriesbeing “struggle,” “learned it,” and “knew it,” each corresponding tocertain algorithmically determined sequences of answers and knowledgestates collected. The struggle bar graph 1850 is also sortable bysimilar categories as the misinformation bar graph 1550, and the oneshown is sorted by worst performers. For this metric, the worstperformers are the ones with the most struggle, as shown by the bargraphs for the first three learners 1851, 1852, and 1853. The learner1857 who is fourth from the bottom had a much larger percentage oftopics categorized as “learned it” than any of the three worstperformers 1851, 1852, and 1853, which reflects that in this metric,struggle itself is the most important thing to identify. This rankingindicates that not having previous knowledge, but learning it withoutstruggle, is fine, but struggling to learn is a metric that requiresextra attention.

FIG. 19 shows a heat map display 1900 that provides a more detailedbreakdown of metrics of course struggle shown in the course struggledashboard 1800. This heat map display 1900 is similar to the heat mapdisplay 1700 of FIG. 17 , which showed course misinformation. In manyembodiments, the colors (or patterns) of each range of struggle may bedifferent from the colors or patterns used in the course misinformationheat map display 1700, so an administrator can easily differentiatebetween the two. An administrator can use the heat map display optionicons 1910 and the bar graph display option icon 1915 to switch betweenthe views shown in FIGS. 19 and 18 . The heat map display 1900 alsocomprises an indicator 1920 and a show/hide adjustment bar 1925 to allowan administrator to view only certain levels of struggle of interest. Asshown, the indicator 1920 is placed to correspond with a level ofstruggle only above 0.2. The numerical measurement of struggle is lessthan one (i.e., 0, 0.2, 0.4, etc.) because the measure is a count ofnon-progress attempts. Typically, a learner will make only one or twonon-progress attempts, and these attempts may be spread out over severalquestions on the same topic. A learner is likely to not struggle onevery single question in the topic. This placement of the indicatormakes visual blocks depicting “no struggle” or “very little struggle”disappear, leaving only visual indicators of the topics with the moststruggle.

Another aspect of the system of the present disclosure is thatindividual learner reports that show specific answer patterns may becompiled and presented in easy-to-use formats. FIG. 20 shows a screencorresponding to the menu option “Actionable Analytics.” Theseactionable analytics reports show the actual answer and knowledge statepatterns given by a user which resulted in the categorizations of“misinformation,” “struggle,” or both. As shown, the actionableanalytics report view 2000 shows a list of learners with a first learner2001 at the top. The first learner 2001 has a parenthetical description2002 which shows what question, and out of how many, is being referredto for this particular learner. For this first learner 2001, this isquestion “1 out of 1” for which there is a performance concern such asmisinformation and struggle. The previous dashboard views in FIGS. 15-19show color-coded and numerical representations of these performanceissues. As previously discussed, though, the categorizations ofmisinformation and struggle can result from a variety of answer andknowledge state combinations. Providing an entire sequence of answercombinations in a spreadsheet or other report could easily be toovoluminous and cumbersome to be useful. However, the sequence of answercombinations just for the most troublesome questions may be extremelyuseful to a manager or learning administrator.

FIG. 20 shows an answer sequence 2003 for the first learner 2001, whoanswered the same question sure and incorrect three times in a rowbefore answering partially correct (i.e., he picked two unsure answers)and then finally answering sure and correct. This particular answersequence indicates both misinformation (at least one sure and correctanswer) and struggle (more than one incorrect answer). Some learners onthis list may have answer sequences that were troublesome for justmisinformation, or just struggle. Some embodiments may allow theadministrator to sort for either just misinformation or just struggle.Below the answer sequence 2003, the question 2004, answer 2005, andlearning material 2006 are reprinted. The actionable analytic reportview 2000 in FIG. 20 shows just a top section of a report comprisingdetails on multiple learners. Though the first learner has only oneproblematic answer sequence, other learners may have several, and foreach, the answer sequence, question, answer, and learning material arereproduced. This report allows an administrator to view, print, and sendjust the most relevant insights about a learner's problematic assessmentexperience. A manager or instructor can then use this report to reviewwith the learner, and efficiently address misinformation or struggle. Abenefit of this report view is that it extracts and presents, out of alarge database, exactly the information a manager and learner wouldneed.

The display and report generation system described herein may beimplemented via a distributed network and database system 2100 as shownin FIG. 21 . In many embodiments, the display and report generationsystem may be implemented as software-as-a-service, the programs ofwhich may be hosted at hosting servers 2110. The system 2100 providesseveral user interface dashboards, including an authoring interface2112, an administrator interface 2114, a learning interface 2116, and amobile interface 2118. The features and functions of the administratorinterface 2114 and the learning interface 2116 have been described indetail throughout the disclosure. The authoring interface 2112 isconfigured to allow an author to write the questions, answers andlearning materials shown and described throughout the figures. It isalso configured to allow the author to view question feedback shown inFIG. 7 , and to edit questions and answers based on such feedback andrecognized patterns of learner misinformation or struggle. That is, ifan author recognizes, through the course struggle dashboard, coursemisinformation dashboard, or actionable analytics reports thatperformance may be adversely impacted by a poorly written question,answer, or learning material, the author can edit it through theauthoring interface 2112. The mobile interface 2118 may allow any typeof user, including an author, administrator, or learner to view andinteract with any of the authoring, administration, or learningdashboards in a mobile computing device format. The reporting interface2119 displays the various dashboards shown and described throughout thedisclosure, including the course misinformation dashboard and coursestruggle dashboard.

The displays and functions provided through the interfaces are providedwith service-oriented software implementing various algorithms describedthroughout this disclosure, as depicted in the services/algorithmscomponent 2200. These services and algorithms derive their outputs fordisplay on the interfaces through inputs from the transactional database2130. The transactional database 2130 collects inputs from the learninginterface 2116 and stores raw data collected therefrom, such as learneranswers, learner identification, time taken to complete questions ormodules, orders of answers, etc. The data in the transactional database2130 is used by the services/algorithm component 2120 to providefunctions back to the users. For example, the answers given by a learnerare used to determine when to show learning material, when to repeat aquestion, and what order to present the subsequent questions.

However, the data as stored in the transactional database 2130 is not ina particularly usable format for reporting purposes. Therefore, thesystem utilizes ETL (extract, transfer, load)/Data Manipulationcomponent 2132 which takes the data stored in the transactional databaseand transforms it into usable data sets for reporting. This ETL/DataManipulation component 2132 performs functions such as rolling upparticular data sets, creating averages, extracting identified metricsof interest, etc. This manipulated data in then stored in the Data Martcomponent 2134, which may also be referred to as a reporting database.The Data Mart component 2134 stores reporting data in a usable formatfor the reporting features of the present disclosure. The analyticscomponent 2136 applies algorithms for determining what information willbe displayed in various reports. Data is applied through the analyticscomponent, to produce standard reports 2138. These comprise the reportsshown in the dashboards described throughout the disclosure, includingconfidently held misinformation reports 2140, struggle reports 2142,actionable analytics reports 2144, and other customizable reports 2146.As shown, each of these reports are displayed through the reportinginterface 2119. Aspects of the reports are also made available to theother interfaces through certain services applications in theservices/algorithm component 2120. For example, parts of the reportinginterface may be displayed through or incorporated into theadministrator interface 2114.

Another function of the system is that the database 2130 further canexport particular “learning events” 2150 to interfaces with externallearning management systems. Learning events may include certainmilestones or metrics that are helpful for a manager or administrator tobe alerted to when they occur. These can include events such as when alearner completes a module, or when a learner retakes a module, or whenall assigned learners have completed a module. Notifications of theseevents may be actively pushed out through learning management systems.There are a number of learning management systems in the industry, andthe “gradebooks” component 2152 represents an interface (e.g., anapplication program interface) that allows the learning eventinformation 2150 to be transferred to those systems.

The systems and methods described herein can be implemented in a machinesuch as a processor-based system in addition to the specific physicaldevices described herein. FIG. 22 shows a diagrammatic representation ofone embodiment of a machine in the exemplary form of a processor-basedsystem 2200 within which a set of instructions can execute for causing adevice to perform or execute any one or more of the aspects and/ormethodologies of the present disclosure. The components in FIG. 22 areexamples only and do not limit the scope of use or functionality of anyhardware, software, embedded logic component, or a combination of two ormore such components implementing particular embodiments.

Processor-based system 2200 may include processors 2201, a memory 2203,and storage 2208 that communicate with each other, and with othercomponents, via a bus 2240. The bus 2240 may also link a display 2232(e.g., touch screen display), one or more input devices 2233 (which may,for example, include a keypad, a keyboard, a mouse, a stylus, etc.), oneor more output devices 2234, one or more storage devices 2235, andvarious tangible storage media 2236. All of these elements may interfacedirectly or via one or more interfaces or adaptors to the bus 2240. Forinstance, the various non-transitory tangible storage media 336 caninterface with the bus 2240 via storage medium interface 2226.Processor-based system 2200 may have any suitable physical form,including but not limited to one or more integrated circuits (ICs),printed circuit boards (PCBs), mobile handheld devices (such as mobiletelephones or PDAs), laptop or notebook computers, distributed computersystems, computing grids, or servers.

Processors 2201 (or central processing unit(s) (CPU(s))) optionallycontain a cache memory unit 2202 for temporary local storage ofinstructions, data, or computer addresses. Processor(s) 2201 areconfigured to assist in execution of processor-executable instructions.Processor-based system 2200 may provide functionality as a result of theprocessor(s) 2201 executing software embodied in one or more tangible,non-transitory processor-readable storage media, such as memory 2203,storage 2208, storage devices 2235, and/or storage medium 2236. Theprocessor-readable media may store software that implements particularembodiments, and processor(s) 2201 may execute the software. Memory 2203may read the software from one or more other processor-readable media(such as mass storage device(s) 2236, 2236) or from one or more othersources through a suitable interface, such as network interface 2220.The software may cause processor(s) 2201 to carry out one or moreprocesses or one or more steps of one or more processes described orillustrated herein. Carrying out such processes or steps may includedefining data structures stored in memory 2203 and modifying the datastructures as directed by the software.

The memory 2203 may include various components (e.g., machine readablemedia) including, but not limited to, a random access memory component(e.g., RAM 2204) (e.g., a static RAM “SRAM”, a dynamic RAM “DRAM, etc.),a read-only component (e.g., ROM 2205), and any combinations thereof.ROM 2205 may act to communicate data and instructions unidirectionallyto processor(s) 301, and RAM 2204 may act to communicate data andinstructions bidirectionally with processor(s) 2201. ROM 2205 and RAM2204 may include any suitable tangible processor-readable mediadescribed below. In one example, a basic input/output system 2206(BIOS), including basic routines that help to transfer informationbetween elements within processor-based system 2200, such as duringstart-up, may be stored in the memory 2203.

Fixed storage 2208 is connected bidirectionally to processor(s) 2201,optionally through storage control unit 2207. Fixed storage 2208provides additional data storage capacity and may also include anysuitable tangible processor-readable media described herein. Storage2208 may be used to store operating system 2209, EXECs 2210(executables), data 2211, APV applications 2212 (application programs),and the like. Often, although not always, storage 2208 is a secondarystorage medium (such as a hard disk) that is slower than primary storage(e.g., memory 2203). Storage 2208 can also include an optical diskdrive, a solid-state memory device (e.g., flash-based systems), or acombination of any of the above. Information in storage 2208 may, inappropriate cases, be incorporated as virtual memory in memory 2203.

In one example, storage device(s) 2235 may be removably interfaced withprocessor-based system 2200 (e.g., via an external port connector (notshown)) via a storage device interface 2225. Particularly, storagedevice(s) 2235 and an associated machine-readable medium may providenonvolatile and/or volatile storage of machine-readable instructions,data structures, program modules, and/or other data for theprocessor-based system 2200. In one example, software may reside,completely or partially, within a machine-readable medium on storagedevice(s) 2235. In another example, software may reside, completely orpartially, within processor(s) 2201.

Bus 2240 connects a wide variety of subsystems. Herein, reference to abus may encompass one or more digital signal lines serving a commonfunction, where appropriate. Bus 2240 may be any of several types of busstructures including, but not limited to, a memory bus, a memorycontroller, a peripheral bus, a local bus, and any combinations thereof,using any of a variety of bus architectures. As an example and not byway of limitation, such architectures include an Industry StandardArchitecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro ChannelArchitecture (MCA) bus, a Video Electronics Standards Association localbus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express(PCI-X) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport(HTX) bus, serial advanced technology attachment (SATA) bus, and anycombinations thereof.

Processor-based system 2200 may also include an input device 2233. Inone example, a user of processor-based system 2200 may enter commandsand/or other information into processor-based system 2200 via inputdevice(s) 2233. Examples of an input device(s) 2233 include, but are notlimited to, an alpha-numeric input device (e.g., a keyboard), a pointingdevice (e.g., a mouse or touchpad), a touchpad, a joystick, a gamepad,an audio input device (e.g., a microphone, a voice response system,etc.), an optical scanner, a video or still image capture device (e.g.,a camera), and any combinations thereof. Input device(s) 2233 may beinterfaced to bus 2240 via any of a variety of input interfaces 2223(e.g., input interface 2223) including, but not limited to, serial,parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination ofthe above.

In particular embodiments, when processor-based system 2200 is connectedto network 2230, processor-based system 2200 may communicate with otherdevices, specifically mobile devices and enterprise systems, connectedto network 2230. Communications to and from processor-based system 2200may be sent through network interface 2220. For example, networkinterface 2220 may receive incoming communications (such as requests orresponses from other devices) in the form of one or more packets (suchas Internet Protocol (IP) packets) from network 2230, andprocessor-based system 2200 may store the incoming communications inmemory 2203 for processing. Processor-based system 2200 may similarlystore outgoing communications (such as requests or responses to otherdevices) in the form of one or more packets in memory 2203 andcommunicated to network 2230 from network interface 2220. Processor(s)2201 may access these communication packets stored in memory 2203 forprocessing.

Examples of the network interface 2220 include, but are not limited to,a network interface card, a modem, and any combination thereof. Examplesof a network 2230 or network segment 2230 include, but are not limitedto, a wide area network (WAN) (e.g., the Internet, an enterprisenetwork), a local area network (LAN) (e.g., a network associated with anoffice, a building, a campus or other relatively small geographicspace), a telephone network, a direct connection between two computingdevices, and any combinations thereof. A network, such as network 2230,may employ a wired and/or a wireless mode of communication. In general,any network topology may be used.

Information and data can be displayed through a display 2232. Examplesof a display 2232 include, but are not limited to, a liquid crystaldisplay (LCD), an organic liquid crystal display (OLED), a cathode raytube (CRT), a plasma display, and any combinations thereof. The display2232 can interface to the processor(s) 2201, memory 2203, and fixedstorage 2208, as well as other devices, such as input device(s) 2233,via the bus 2240. The display 2232 is linked to the bus 2240 via a videointerface 2222, and transport of data between the display 2232 and thebus 2240 can be controlled via the graphics control 2221.

In addition to a display 2232, processor-based system 2200 may includeone or more other peripheral output devices 2234 including, but notlimited to, an audio speaker, a printer, and any combinations thereof.Such peripheral output devices may be connected to the bus 2240 via anoutput interface 2224. Examples of an output interface 2224 include, butare not limited to, a serial port, a parallel connection, a USB port, aFIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

In addition or as an alternative, processor-based system 2200 mayprovide functionality as a result of logic hardwired or otherwiseembodied in a circuit, which may operate in place of or together withsoftware to execute one or more processes or one or more steps of one ormore processes described or illustrated herein. Reference to software inthis disclosure may encompass logic, and reference to logic mayencompass software. Moreover, reference to a processor-readable mediummay encompass a circuit (such as an IC) storing software for execution,a circuit embodying logic for execution, or both, where appropriate. Thepresent disclosure encompasses any suitable combination of hardware,software, or both.

Those of skill in the art would understand that information and signalsmay be represented using any of a variety of different technologies andtechniques. For example, data, instructions, commands, information,signals, bits, symbols, and chips that may be referenced throughout theabove description may be represented by voltages, currents,electromagnetic waves, magnetic fields or particles, optical fields orparticles, or any combination thereof.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the embodiments disclosed herein may be implemented aselectronic hardware, or hardware in connection with software. Variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or hardware that utilizessoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present invention.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed with a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor may be a microprocessor, but in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

The steps of a method or algorithm described in connection with theembodiments disclosed herein may be embodied directly in hardware, in asoftware module executed by a processor, or in a combination of the two.A software module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in an ASIC. The ASIC mayreside in a user terminal. In the alternative, the processor and thestorage medium may reside as discrete components in a user terminal.

The previous description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the presentdisclosure. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the disclosure. Thus, the present disclosure is notintended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the principles and novelfeatures disclosed herein.

1. (canceled)
 2. A digital tutoring system comprising: a user interfaceconfigured to present a learning module comprising a series of questionsand answers on a plurality of successive question/answer screens of theuser interface, wherein at least one of the plurality of successivescreens comprises a plurality of inputs, wherein each of the pluralityof inputs is associated with an answer to a question, and wherein afirst input action to an input and a second input action to the inputeach indicate a different confidence level of a user's answer; and anartificial intelligence (AI) guidance system configured to process aplurality of first input actions and second input actions of a userprogressing through the learning module to determine a confidencepattern of the user with respect to the learning module, and to generatea personalized prompt based at least in part on the confidence patternto guide the user in real-time through the learning module.
 3. Thedigital tutoring system of claim 2, wherein: the first input actioncauses at least one of the inputs to provide a first visual indicationof the first input action, and the second input action causes the atleast one of the inputs to provide a second visual indication of thesecond input action.
 4. The digital tutoring system of claim 2, wherein:the artificial intelligence (AI) guidance system is configured toprocess the plurality of first input actions and second input actions ofthe user progressing through the learning module to determine aknowledge pattern of the user with respect to the learning module, andto generate the personalized prompt based at least in part on theknowledge pattern.
 5. The digital tutoring system of claim 2, wherein:the artificial intelligence (AI) guidance system is configured toprocess the plurality of first input actions and second input actions ofprevious users who completed the learning module, and to generate thepersonalized prompt for the user based at least in part on thoseprevious users' answer patterns.
 6. The digital tutoring system of claim2, wherein: the artificial intelligence (AI) guidance system isconfigured to process the plurality of first input actions and secondinput actions of the user progressing through the learning module todetermine a behavior pattern of the user with respect to the learningmodule, and to generate the personalized prompt based at least in parton the behavior pattern.
 7. The digital tutoring system of claim 2,wherein: the AI guidance system is configured to process the pluralityof first input actions and second input actions of the user progressingthrough the learning module to determine one or more of: a userengagement assessment with the learning module, a user engagementassessment with generated guidance for the learning module, a userdistraction assessment, a user cheat assessment, and a user struggleassessment; and the AI guidance system is configured to generate thepersonalized prompt based at least in part on one or more of: the userengagement assessment with the learning module, the user engagementassessment with generated guidance for the learning module, the userdistraction assessment, the user cheat assessment, and the user struggleassessment.
 8. The digital tutoring system of claim 2, wherein: the AIguidance system is configured to generate the personalized prompt basedat least in part on feedback from the user or queries by the user. 9.The digital tutoring system of claim 2, wherein: the AI guidance systemis configured to generate the personalized prompt based at least in parton a combination of the confidence pattern and one or more of authorcontent and author metadata applied to the learning module.
 10. Thedigital tutoring system of claim 2, wherein: the personalized promptincludes additional learning material for the learning module havingcognitive or motivational value for the user.
 11. The digital tutoringsystem of claim 2, wherein: the AI guidance system is configured togenerate an alert message for a dashboard of the learning module basedat least in part on one or more determined confidence patterns.
 12. Thedigital tutoring system of claim 2, wherein: the AI guidance system isconfigured to generate the personalized prompt based at least in part ona combination of the confidence pattern and unpublished data fromexperiments in cognitive science.
 13. The digital tutoring system ofclaim 2, wherein the AI guidance system is configured to generate thepersonalized prompt based at least in part on a combination of theconfidence pattern and published data from experiments in cognitivescience.
 14. A method comprising: presenting, with a user interface, alearning module comprising a series of questions and answers on aplurality of successive question/answer screens of the user interface,wherein at least one of the plurality of successive screens comprises aplurality of inputs, wherein each of the plurality of inputs isassociated with an answer to a question, and wherein a first inputaction to an input and a second input action to the input each indicatea different confidence level of a user's answer; processing, with anartificial intelligence (AI) guidance system, a plurality of first inputactions and second input actions of a user progressing through thelearning module to determine a confidence pattern of the user withrespect to the learning module; and generating, with the AI guidancesystem, a personalized prompt based at least in part on the confidencepattern to guide the user in real-time through the learning module. 15.The method of claim 14, wherein: the first input action causes at leastone of the inputs to provide a first visual indication of the firstinput action, and the second input action causes the at least one of theinputs to provide a second visual indication of the second input action.16. The method of claim 14, further comprising: processing the pluralityof first input actions and second input actions of the user progressingthrough the learning module to determine a knowledge pattern of the userwith respect to the learning module; and generating the personalizedprompt based at least in part on the knowledge pattern.
 17. The methodof claim 14, further comprising: processing the plurality of first inputactions and second input actions of previous users who completed thelearning module; and generating the personalized prompt based at leastin part on those previous users' answer patterns.
 18. The method ofclaim 14, further comprising: processing the plurality of first inputactions and second input actions of the user progressing through thelearning module to determine a behavior pattern of the user with respectto the learning module; and generating the personalized prompt based atleast in part on the behavior pattern.
 19. The method of claim 14,further comprising: processing the plurality of first input actions andsecond input actions of the user progressing through the learning moduleto determine one or more of: a user engagement assessment with thelearning module, a user engagement assessment with generated guidancefor the learning module, a user distraction assessment, a user cheatassessment, and a user struggle assessment; and generating thepersonalized prompt based at least in part on one or more of: the userengagement assessment with the learning module, the user engagementassessment with generated guidance for the learning module, the userdistraction assessment, the user cheat assessment, and the user struggleassessment.
 20. The method of claim 14, further comprising: generatingthe personalized prompt based at least in part on feedback from the useror queries by the user.
 21. The method of claim 14, further comprising:generating the personalized prompt based at least in part on acombination of the confidence pattern and one or more of author contentand author metadata applied to the learning module.
 22. The method ofclaim 14, wherein: the personalized prompt includes additional learningmaterial for the learning module having cognitive or motivational valuefor the user.
 23. The method of claim 14, further comprising: generatingan alert message for a dashboard of the learning module based at leastin part on one or more determined confidence patterns.
 24. The method ofclaim 14, further comprising: generating the personalized prompt basedat least in part on a combination of the confidence pattern andunpublished data from experiments in cognitive science.
 25. The methodof claim 14, further comprising: generating the personalized promptbased at least in part on a combination of the confidence pattern andpublished data from experiments in cognitive science.